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Conclusions 


In this study, we coupled social and emotional 
learning and learning analytics in four undergraduate 
courses making extensive use of the LMS and 
examined relationships among social and emotional 
(SE) skills, behaviors recorded in the LMS, and 
course grades. Our findings showed that SE skills, 
LMS data, and grades are largely associated with 
one another in expected ways, with course grades 
being robustly correlated with LMS behaviors across 
all courses, while different sets of LMS behaviors 
correlated significantly with different SE skills. 
Implications of results, limitations, and future work 
are discussed. 


So What? 


Student interactions with learning management 
systems (LMS) and other educational technologies 
provide detailed information about how students 
interact with learning resources and activities. 
Learning analytics researchers and educational 
technologists have demonstrated that these data can 
be used to predict student course grades. However, 
much less is known about the psychological 
constructs that underlie student online behaviors, 
which are needed to help faculty, advisors, and 
students themselves interpret predictions and 
improve learning outcomes. 


Now What? 


While educational psychologists have found that self- 
reported social and emotional skills have significant 
relationships with course grades, little is known about 
how these skills are enacted in authentic learning 
contexts. This study and others examining behavioral 
residue offer promise for unobtrusive SE skill 
assessment with high validity. 
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Introduction 


A focus of learning analytics research has been the collection, analysis, and 
interpretation of data from online academic technologies, particularly in higher education 
where these technologies are incorporated extensively into the student learning 
experience. In particular, the learning management system (LMS) has emerged 

as a technology frequently integrated into all courses, whether online, in person, 

or hybrid, with three-quarters of students using the LMS in all or nearly all of their 
courses (Galanek, Gierdowski, & Brooks, 2018). Since the 1990s, learning analytics 
researchers and educational technologists have been creating predictive models of 
student academic performance based on “clickstream” data, which are generated by 
student interactions with LMS resources and activities (Wang & Newlin, 2002). Despite 
the ultimate goal of improving student learning (vs. simple prediction), there are few 
empirical studies of intervention efforts. A recent systematic review identified only 11 
published studies of interventions, and those studies were of mixed quality and had 
inconsistent findings (S@nderlund, Hughes, & Smith, 2019). 


The scarcity of intervention studies is due to many factors; among them is a lack of 
attention to the underlying reasons for the captured behaviors in clickstream data. This 
issue has become known as the “click to constructs” problem (Knight & Buckingham 
Shum, 2017). Researchers, especially those coming from an educational psychology 
background, have begun to articulate constructs potentially related to clickstream data. 
To date, self-regulated learning has been one of the focal constructs of investigation 
(Roll & Winne, 2015). It is important to focus on such constructs because they are 
related to the skills and strategies required of students to be effective in their courses. 
They are linked with study habits and systematic behaviors rather than students’ 

prior Knowledge in the particular area of study or mere quantity of activity, which risks 
reducing analytics to a behaviorist educational theory in which more activity is equated 
with more learning (but of course, there could be many activities with no actual learning 
involved). Given that higher education generally requires greater initiative on the part of 
the student (e.g., students may be required to access course materials and activities on 
their own volition and, to some degree, on their own schedule), self-regulated learning 
is a reasonable starting point for this research. In this study, we extend this research to 
consider constructs from the area of social and emotional learning. 


Several social and emotional skills (SE skills) have been identified in the literature 

as relevant for success in school, work, and life (Roberts, Kuncel, Shiner, Caspi, & 
Goldberg, 2007), and it has been noted that these findings generally replicate (Soto, 
2019) and generalize across different samples (Soto, 2020). There is ample evidence, 
for example, that sustaining effort-related skills (i.e., skills related to being organized, 
persistent, reliable, etc.) are associated with academic success at all levels of education 
and are even on par with cognitive ability in predicting grades in secondary and tertiary 
education levels (Poropat, 2009). Given their importance for academic success, 
assessment of students’ SE skills is invaluable, and the LMS and other academic 
technologies offer the opportunity for unobtrusive assessment at an unprecedented 
scale. Rather than tasking students with engaging in additional assessment and relying 
on self-reports which may be limited by issues such as response manipulation or 
response biases, SE skills may be indirectly observable through LMS clickstream data. 
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The practice of observing psychological constructs through such digital footprints 

is becoming more common, allowing researchers to examine so-called “behavioral 
residue,” which is the physical trace of activity in an environment (Gosling, Ko, 
Mannarelli, & Morris, 2002), often an environment with high ecological validity 
(Gladstone, Matz. & Lemaire, 2019), which can give insight into a person’s 
characteristics such as SE skills. This has been carried out in various settings including 
Facebook (Kosinski et al., 2015), music collections (Nave et al., 2018), and spending 
records (Gladstone et al., 2019). We are interested in examining the digital footprint in an 
academic setting to examine associations among SE skills, LMS behaviors, and course 
grades. While we know that SE skills are related to course grades, we do not understand 
the mechanisms by which positive SE skills lead to improved school outcomes. For 
example, we would expect that having attention to detail, being organized, and goal 
striving (i.e., having high levels of sustaining effort) would lead to better course grades 
through better study habits. That is, individuals with higher levels of sustaining effort may 
be more likely to make use of time-sensitive LMS features, which in turn, results in better 
course grades. We do not know, however, whether SE skills are expressed through LMS 
behaviors. In this study, we explore this possibility by examining relationships among SE 
skills, behaviors recorded in the online learning environment, and course grades. We 
carry out this study in four separate courses at a single higher education institution. 


Method 


Participants 


Participants were undergraduate students at a mid-sized public research university on 
the East Coast. Students (N = 1,427) from four courses during the spring 2019 semester 
participated in the study. The sample was diverse with respect to gender, race/ethnicity, 
first-generation student status, and transfer student status; demographic information, 
which was extracted from the Peoplesoft student information system, is provided in 
Table 1 for students in each course. The four courses were web enhanced with varying 
degrees of technology and LMS integration. Students opted into the study with an 
incentive of extra course credit. 


Data Sources and Measures 
Course Syllabi and Instructor Interviews 


The course syllabi were reviewed, and relevant dates and learning activities were 
identified. The faculty members then took part in individual semi-structured interviews. 
They were questioned about their use of the LMS for the course activities and their 
experience pertaining to which student behaviors are typically associated with course 
grades. This information was used to create a preliminary map of features that could be 
extracted from the LMS data. Once the LMS data were received, the mappings were 
discussed with the faculty members and an academic technology administrator to ensure 
that they were accurate. The courses varied in terms of their use of the LMS, both in 
terms of the features used and the frequency of use (see Tables 2-5). 
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Blackboard Data (LMS) 


The quantitative data analyzed include the activity log data extracted from the 
Blackboard Learn LMS using data provided by the Blackboard Data product. Activity 
measures were provided at the individual student-activity level for all interactions 

with course materials and activities that were recorded in the database, with a total 

of approximately 1.6 million records across the four courses (approximate values 

per course were 700,000 for Chemistry, 100,000 for Math, 375,000 for Physics, and 
450,000 for Psychology). We only included LMS behavioral data from the activity table; 
in-course graded items were excluded as these items are not independent from final 
course grade. 


ACT Tessera College 


ACT® Tessera® (ACT, 2020) is a SE skills assessment measuring five skills, which 
are listed and described in Table 6. Students’ skills are assessed via three methods 
— Likert items, forced choice items, and situational judgment tests. This multi-method 
approach avoids the pitfalls associated with any single item type (Kenny & Kashy, 
1992), such as the ease with which test-takers can fake their responses to Likert 
items (Zickar, Gibby, & Robie, 2004). Responses from these three item types are 
then aggregated into a multidimensional IRT score for each skill (Anguiano-Carrasco, 
Walton, Murano, Burrus, & Way, 2017), which roughly fall between +3. The college 
version of ACT® Tessera® is not yet operational but has undergone pilot studies to 
establish its reliability and validity. 


Procedure 
General Study Procedure 


The study was reviewed by the university’s Institutional Review Board. All student 
data were anonymized by using a random identifier generated for this study, and no 
personally identifiable information was included in the dataset. The school’s data 
privacy policies, which are disclosed to students and posted in the student portal, 
allow for the analysis of student learning activity data for the purpose of educational 
improvement without explicit consent. The instructors introduced the complete study to 
their classes, and interested students opted in to participate in the assessment of SE 
skills. Participating students were given extra credit points for the course. As a result, 
all student data were included in analysis that required only LMS use, and for analysis 
of SE skills, the subset of students that completed the assessment were included (we 
had ACT Tessera data for 68% of the total sample). 


LMS Clickstream Data Processing 


The online interaction features were generated from the LMS clickstream data by 
uniformly formatting log events (Greene et al., 2019) in which clickstream data were 
manually mapped to learning activities as determined through analysis of the syllabus 
and instructor semi-structured interviews. After manual inspection, we determined 
that the action field alone (e.g., “opened”) was insufficient and needed to be joined 
with the label of the item that the action was taken in reference to, which was a 
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complex pairing. For example, course item values in the LMS data include “Week 12: 
Electrochemistry” or “CHEM 102 Practice Exam 4B,” which were easily interpretable 
from the course syllabus, while others needed confirmation from the instructor (e.g., 
“2/27 CL” and “18.7 RQ”). Hence, we created broader activity categories (referred to 
interchangeably as “activity”) for these activity measures in the LMS data using the 
course syllabi with confirmation from the instructors, which resulted in 583 unique 
activity measures in the LMS data that were recoded to a total of 35 activity categories 
(Tables 7-10). 


Several data transformations were conducted to remove irrelevant data and to control 
for inaccurate session duration recordings. Activities with a duration of zero seconds 
were removed as they represented navigation or system activity. The values of 
activity duration and session (login) ID were also recalculated to control for the known 
challenge of extracting this type of information from log information. Each activity was 
truncated to a maximum length of 30 minutes, and a new session was calculated if 
two hours or more had lapsed between student actions to adjust for students that had 
ceased using the system but had not logged out. These calculations were made after 
reviewing the data distributions for empirically justified threshold levels. 


We initially developed a design pattern that used evidence-centered design principles 
(Mislevy, Almond, & Lukas, 2003) to create indicators from the LMS data (Beheshitha, 
GaSevic, & Hatala, 2015) that best described the SE skills. Evidence-centered design 
is an assessment design process providing the language, concepts, and knowledge 
representations to build an evidentiary argument for that particular assessment. A 

key step in this process is domain modeling that builds a design pattern including the 
student model, task model, and evidence model. The student model may indicate the 
knowledge, skills, abilities, and behaviors being measures (e.g., sustaining effort). The 
task model would indicate which activities may elicit the knowledge, skills, abilities, and 
behaviors (e.g., taking an online practice exam). The evidence model indicates what 
data provide evidence of those knowledge, skills, abilities, and behaviors (e.g., getting 
a question correct after multiple incorrect attempts). 


Despite undertaking this process for the SE skills, the information captured in the LMS 
activity measure log proved to be inadequate in generating the needed interaction 
features based on the design patterns. Therefore, from the reformatted LMS data, we 
were able to compute student-level interaction features that described access to these 
activities (LMS activity features). These were then merged to the individual student’s 
SE skill scores, final course grade, and demographic information. This process 
required extensive manual interpretation and recoding. 


From the LMS log data, relevant information about a student accessing a particular 
activity included the session ID it belonged to, the duration of that activity in a 

session, and number of instances of that activity in a session. Therefore, for each 
activity category, we computed its average measures of engagement across all of a 
student’s recorded sessions. These calculations included three average measures: 
average count or number of attempts at an activity in a session (activity appended 
with “Attempt”), average time spent or total duration for an activity in a session (activity 
appended with “TimeSpent”), and average variability of time spent or total duration 

for an activity in a session (activity appended with “TimeSpentSD”). This approach 
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to summarizing features was used due to its success with similar problems in other 
learning systems (cf. Baker et al., 2012; Baker, Goldstein, & Heffernan, 2011; Pardos 
et al., 2014). We accounted for the number of sessions when looking at counts and 
duration of an activity, instead of count and sum across the entire semester, to have a 
more nuanced measure of activity engagement. For example, throughout the semester, 
a student accessed an assignment activity in the LMS across three login sessions, and 
for each session, the student accessed it twice. Thus, the feature assignment_Attempt 
for this student would be equal to 2. If we included the number of sessions a student 
had as an interaction feature, there were a total of 106 unique LMS activity features 
across all students in all four courses. 


Results 


Associations Among SE Skills, LMS Behaviors, and Course 
Grades 


After generating the features at the student level, we examined the correlations among 
the LMS behaviors, SE skills, and course grade. Here we present the findings course 
by course. Any correlations mentioned or presented in the tables are statistically 
significant (p < .05; see Tables 7-10). Descriptive statistics for grades and SE skills 
across courses are provided in Table 11. 


Chemistry 


Course grade had the strongest relationship with Sustaining Effort (r= .34), as 
expected, and was also significantly correlated with Maintaining Composure (r= .16), 
Keeping an Open Mind (r= .14), and Social Connection (r= .12). 


Of the four courses, Chemistry made the most use of the LMS. A total of 44 LMS 
activity measures were found to be significantly correlated with grade or one or more 
SE skills. Of these 44 features, 32 were associated with grade. Grade was most 
strongly related to activity measures involving discussion board usage, specifically 
number of attempts on the board (r= .36), time spent on the board (r= .32), and 
variability of time spent on the board (r= .31). Of the 32 features, only number of 
attempts on syllabus was in the negative direction. 


With regard to the SE skills’ associations with LMS activity measures, as expected, 
Sustaining Effort had the most statistically significant correlations (with 25 LMS 
activities), followed by Getting Along with Others (16), Keeping an Open Mind (7), 

and Maintaining Composure and Social Connection (both with 4). All associations 
were in the positive direction with just two exceptions—Maintaining Composure- 
activity correlation and Social Connection-activity correlation. Sustaining Effort was 
most strongly associated with activity measures related to assignments (r= .23), 
specifically variability of time spent on assignment (r= .23), time spent on assignment, 
and number of attempts on assignment (r= .21). The LMS activity measure that was 
related to the most SE skills (4) was variability of time spent on exam review materials. 
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Math 


The only SE skill that was significantly correlated with course grade was Keeping an 
Open Mind, and it was not in the expected direction (r= -.22). 


Integration of the LMS into the Math course was minimal compared to the other three 
courses. A total of 27 LMS activity measures were found to be significantly correlated 
with grade or one or more SE skills. Of these 27 features, 16 were associated with 
grade. Grade was most strongly related to number of LMS sessions (r= .39). Of the 16 
features, only number of attempts on information was in the negative direction. 


With regard to the SE skills’ associations with LMS activities, Getting Along with 
Others (8) had the most significant correlations with LMS activities, followed closely 
by Sustaining Effort (7) and Keeping an Open Mind (6). Social Connection and 
Maintaining Composure were significantly correlated with only three and two activity 
measures, respectively. Most associations were in the positive direction with a few 
exceptions. The LMS activity that was related to the most SE skills (4) was time spent 
on general content. 


Physics 


Course grade had the strongest relationship with Sustaining Effort (r= .19), as 
expected. 


A total of 31 LMS activity measures were found to be significantly correlated with 
grade or one or more SE skills. Of these 31 features, 16 were associated with grade. 
Grade was most strongly related to number of LMS sessions (r= .36). Fourteen of the 
features were in the positive direction. 


With regard to the SE skills’ associations with LMS activities, Sustaining Effort (18) had 
the most significant correlations with LMS activities, more than double the number for 
any other SE skill. All associations were in the positive direction. No single LMS activity 
was associated with more than three SE skills. 


Psychology 


Counter to expectations, Maintaining Composure and Social Connection were both 
negatively correlated with course grade (r= -.19), and Sustaining Effort was not 
significantly related to grade. 


A total of 50 LMS activity measures were found to be significantly correlated with grade 
or one or more SE skills. Forty of the 50 features were associated with grade, and all 
were positively correlated. Grade was most strongly related to number of LMS sessions 
(r= .53). 


With regard to the SE skills’ associations with LMS activity measures, Sustaining Effort 
(12) and Keeping an Open Mind (10) had the most significant correlations with LMS 
activities, and no other skill was related to more than three activities. Most associations 
were in the positive direction, but there were some exceptions. No single LMS activity 
was associated with more than three SE skills. 
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Discussion and Conclusion 


This study capitalizes on learning analytic techniques to analyze students’ clickstream 
data and link those data to student performance as well as important SE skills, 

which are of ever-increasing focus in research (e.g., Taylor, Oberle, Durlak, & 
Weissberg, 2017) and policy (e.g., Dusenbury & Weissberg, 2018). We examined 
relationships among SE skills, behaviors recorded in the LMS, and course grades in 
four separate university courses making extensive use of an LMS to supplement in- 
person instruction. This study allows us to explore SE skills in relation to the “click to 
constructs” problem (Knight & Buckingham Shum, 2017), which pertains to the lack of 
attention given to the underlying reasons for observed behaviors in clickstream data. 
With the current study, we extend early research focusing on self-regulated learning 
(Roll & Winne, 2015) to consider a broader set of skills. Overall, our findings suggest 
that SE skills, LMS data, and grades are largely associated with one another in 
expected ways. We review our findings below and discuss their implications for SE skill 
assessment and interventions. 


Associations Among SE Skills, LMS Behaviors, and Course 
Grades 


As expected, LMS behaviors and course grades were robustly correlated, which 

is intuitive given the direct relationship between the two; that is, if students are not 
engaged with class activities and content, their grades should be directly affected in 
a negative manner. To some degree, the expected associations between SE skills 
and grades were observed, but not across all SE skills and courses. In Chemistry 
and Physics, Sustaining Effort had the strongest correlation with grade, which was 
expected. In Math, Keeping an Open Mind had the strongest correlation with grade, 
but it was a negative association, which counters what Poropat (2009) reported in his 
meta-analysis. In Psychology, Social Connection and Maintaining Composure had 
the strongest correlations with grade, and although the negative direction of those 
correlations was consistent with Poropat’s results, our estimates were considerably 
greater in magnitude. 


Of particular importance for this study, however, we found that SE skills have 
systematic relationships with LMS behaviors when analyzed in relation to individual 
activities. Differences across the SE skill-LMS activity correlations provide evidence 

of the validity of the SE skills; that is, different SE skills are associated with different 
student actions. As anticipated, Sustaining Effort was associated with the greatest 
number of LMS activities and to the greatest degree, speaking to the tendency 

of students with a high level of Sustaining Effort to attend to detail, be organized, 

and persist at completing their work, all of which are important characteristics for 
succeeding in an asynchronous online learning environment. As another example, in 
Chemistry, Keeping an Open Mind was the only SE skill significantly related to activities 
pertaining to learning objectives, which may reflect the tendency of students with a high 
level of Keeping an Open Mind to be highly inquisitive and think about high-level ideas, 
such as how specific learning materials relate to the overarching learning objectives. 
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Differences Across SE Skills, LMS Behaviors, and Courses 


There were some differences observed across variables or courses that warrant 
consideration. First, some LMS behaviors appeared to be highly related to grade but 
not SE skills, for example. A concrete illustration of this is in the Chemistry course 
where activities related to discussion board usage were the behaviors most strongly 
associated with grade but were only associated with one SE skill (Sustaining Effort), 
and to a relatively small degree. This is an area in which the faculty interviews had 
tremendous value. The instructor of this course explained that the discussion forum 
is infrequently used in the course, but the students who do participate in the forum 
are the more proactive students who point out errors in the book and ask targeted 
and insightful questions. Second, some courses had many more significant grade- 
LMS activity or grade-SE skill correlations than others (e.g., Psychology had nearly 
twice the number of significant correlations between LMS activities and grade or 

SE skills as Math) or had different significant associations (e.g., in Physics, one SE 
skill was significantly correlated with number of sessions in the LMS, whereas three 
SE skills were significantly correlated with number of session in the LMS in Math). 
Such inconsistencies can be attributed to the varying designs and requirements of 
the courses. Not all courses used the LMS with the same frequency or in the same 
manner. The courses were not designed with a priori mappings of LMS behaviors to SE 
skills at all, let alone consistently. 


Limitations and Future Work 


This project was largely exploratory and was conducted at a single university in a 
limited number of courses, and as such, the findings may not generalize across area 
of study; for example, we had an oversampling of STEM classes in this study. There 
are mean-level differences in SE skill-related constructs across majors (Vedel, 2016), 
and there are possibly differences in the variability of those constructs across majors 
as well. With differences in variability, it is possible that SE skills may relate differently 
with grades and LMS behaviors across diverse courses. For example, courses in the 
arts may have greater effects related to Keeping an Open Mind. Differences in student 
characteristics across various universities could potentially affect findings as well 
(although we found no significant subgroup differences in our analyses). As another 
point of consideration for future study, analyses should include a threshold requirement 
for inclusion in such studies. For example, courses with low usage of the LMS will 
naturally have fewer observed LMS-grade and LMS-SE skill associations. Finally, the 
manual mapping of clickstream log data to course-specific features limits the potential 
to scale out this work without a significant amount of additional effort. Automated 
methods to identify features that also have fidelity with the underlying course design 
would enable broader testing and deployment of models. 


Implications 


The findings presented above have a number of important implications for practice. 
This work represents an important example of examining behavioral residue (Gosling 
et al., 2002) in an environment with high ecological validity (Gladstone et al., 

2019). This method of unobtrusively observing students’ SE skills in the real world 

in real time could be invaluable. The LMS and other academic technologies offer 

the opportunity for SE skill assessment at an unprecedented scale. In addition to 
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allowing for large-scale investigations, this also alleviates a big problem in education, 
particularly in primary and secondary school, which is too much time spent on testing. 
An overwhelming majority (81%) of teachers feel as though their students spend too 
much time taking mandated tests (Rentner, Kober, Frizzell, & Ferguson, 2016), and the 
sentiment is surely echoed in the student body. Not only can testing be burdensome, 
but conventional methods of assessing SE skills are subject to various biases and 
limitations (Wetzel, Bohnke, & Brown, 2016), which may threaten the validity of the 
assessments. This study and others examining behavioral residue offer promise for 
unobtrusive SE skill assessment with high validity. 
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Table 1. Sample Demographic Data 


Chemistry Physics Psychology 
Variable % % % 
Female 242 50 74 36 195 61 25) 40 
Race / Ethnicity 
Asian 141 29 66 32 117 37 86 27 
Black / African American 84 17 52 25 61 19 70 22 
Hispanic / Latino 43 9 11 is 19 6 25 8 
White 189 39 60 29 105 33 106 34 
Two or More 29 6 11 5 10 3 19 6 
Not specified 3 1 6 3 6 2 7 2 
Sedcaee nk College 19-24 65 31 86 27 79 25 
Transfer Student (yes) 53 11 89 43 101 32 47 15 
ACT Tessera Participant (yes) 406 83 104 50 223 70 We 515) 
Total 489 207 319 314 


Table 2. Distribution of LMS Activities’ Total Semester Count: Chemistry 


AVfelar=Ve)(= M Vola) psy B) Min Max 
Number of sessions 168.03 162 56.91 18 383 
Announcement 267.52 242 131.64 27 1400 
Answer key 0.60 0 1.94 0 20 
Assignment 256.21 255 113.86 8 893 
Discussion board Som 24 99.55 0 858 
Discussion class 4.53 3 4.88 0 46 
Email / Messages PA soit 1 3.24 0 24 
Exam review materials 39.24 36 22.86 0 144 
General content 187.16 187 39.60 26 366 
Gradebook 60.29 44 67.31 3 986 
Help resource 3.91 2 5.04 0 45 
Information 4.99 0 13.43 0 107 
Learning objectives 0.34 0 leas 0 ‘ffl 
Lecture/Class Activities 13.65 13 7.04 0 61 
Orientation 4.47 4 3.39 0 24 
Practice assignment 110 97 82.73 0 524 
Practice exam 55.64 48 39.40 0 314 
Reading quiz 173.04 171 44.42 7 441 
Remediation/ Makeup 77.92 UU 16.02 17 152 
Survey 23.15 24 9.09 0 55 
Syllabus 26.69 i) 24.67 (| 199 
Textbook 0.88 0 19.35 0 428 


Note. n = 489 students with course grade 
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Table 3. Distribution of LMS Activities’ Total Semester Count: Math 


Wfelatele)(= 

Number of sessions 
Announcement 
Answer key 
Assignment 
Discussion board 
Email / Messages 
Exam 

Exam review materials 
General content 
Gradebook 
Groupwork 

Help resource 
Information 

Learning objectives 
Lecture notes 
Lecture/Class Activities 
Orientation 

Reading quiz 
Remediation/ Makeup 
Syllabus 

Textbook 

Worksheet 


M 
96.50 
61.21 

1.77 

2.69 
4.82 
0.43 
0 
41.15 
Zoroo) 
37.28 

0.01 

0.02 
32.50 

0.05 
19.69 

0.17 
63°95 

4.36 
4.39 
23.10 
62.01 
25.82 


Note. n = 207 students with course grade 
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Table 4. Distribution of LMS Activities’ Total Semester Count: Physics 


WV/eVatele)(= 

Number of sessions 
Announcement 
Answer key 
Discussion board 
Email / Messages 
Exam 

Exam questions 
Exam review materials 
General content 
Gradebook 
Groupwork 

Help resource 
Information 

Lab activities 

Lab exam 

Lecture notes 
Lecture/Class Activities 
Orientation 

Reading quiz 
Syllabus 

Textbook 


43.11 
15 

36 
201.87 
1.91 
240.29 


Note. n = 319 students with course grade 
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Table 5. Distribution of LMS Activities’ Total Semester Count: Psychology 

Wfelat=le)(= M WV Lola) psy) Min Max 
Number of sessions 81.09 68 58.84 1 368 
Announcement 52.89 13.50 92.24 0 642 
Chapter exam 343.70 Si7/ 111510) 100.85 0 616 
Chapter readings 66.76 61 40.16 0 308 
Clicker OR75 0 1.61 0 12 
Discussion board 6.75 0 13.86 0 102 
Email / Messages 0.36 0 1.12 0 8 
Exam 33.02 30.50 16.54 0 90 
Extra credit 1.28 0 3.40 0 29 
General content 163.83 131 123.35 0 936 
Gradebook 30.26 12 49.95 0 385 
Groupwork 0 0 0.06 0 1 
Help resource 6.09 4 6.37 0 37 
Information 23.32 13 30.28 0 209 
Lab activities 115.67 109.50 62.30 0 530 
Lab assignment 64.52 60.50 42.31 0 307 
Learning objectives 0.04 0 0.25 0 3 
Lecture/Class Activities 0.03 0 0.22 0 2 
Online quiz 16.69 14 11.91 0 74 
Orientation 70.80 64 48.03 0 400 
Reading quiz 228.05 230 28.25 0 621 
Syllabus 13.83 11 10.76 0 84 
Textbook 137.84 40.50 293.89 0 3220 


Note. n = 314 students with course grade 


Table 6. Social and Emotional Skills Assessed with ACT Tessera 


Description 


Reflects the extent to which a student’s actions demonstrate... 


... persistence, goal striving, reliability, dependability, and attention to 
detail at school. 


Sustaining Effort 


Getting Along with Others 
Maintaining Composure 


... collaboration, empathy, helpfulness, trust, and trustworthiness. 


... stress management, emotional regulation, a positive response to 
setbacks, and poise. 

Keeping an Open Mind ... creativity, inquisitiveness, flexibility, open-mindedness, and 
embracing diversity. 


Social Connection ... assertiveness, influence, optimism, and enthusiasm. 
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Table 7. Correlations Between LMS Activities and Grade and SE Skills: Chemistry 

LMS Activity Grade Ss) GA MC KOM 

N attempt: Discussion board .36 ath 

T spent: Discussion board B82 13 

Var of T spent: Discussion board ESI IZ 

Number of Sessions 31 .18 14 

T spent: Lecture class activities 24 sili 10 

T spent: Announcements 22 .14 11 

Var of T spent: Announcements AA ml Bal 

T spent: Gradebook 19 4 

Var of T spent: Lecture class activities 19 212 

Var of T spent: Information .18 

Var of T spent: General content .16 ld 

Var of T spent: Exam review materials .16 14 At 10 

N attempt: Announcements mll@ ella 14 li 

Var of T spent: Practice exam .15 10 

T spent: Practice exam mls 

N attempt: Information 5 

T spent: Exam review materials 14 lS) 

N attempt: Practice assignment 14 

Var of T spent: Survey 13 

Var of T spent: Assignment =13 23 10 

T spent: General content mil 10 

T spent: Reading quiz 13 

T spent: Information mill 212 

Var of T spent: Reading quiz 13 10 10 

N attempt: Exam review materials ANZ 

T spent: Assignment AZ 23 14 

N attempt: Practice exam alt] 

T spent: Orientation 11 14 3 

N attempt: Survey ail 10 2 

T spent: Syllabus 10 10 

N attempt: General content .09 .10 

N attempt: Syllabus -.11 

N attempt: Assignment | 12 

T spent: Email messages 13 .16 14 

Var of T spent: Orientation mill oth 

T spent: Practice assignment 2 

N attempt: Orientation mall 

N attempt: Gradebook aM 2 

N attempt: Email messages .10 

T spent: Discussion class -.13 

Var of T spent: Learning objectives oS 

N attempt: Learning objectives .10 

T spent: Learning objectives 10 


T spent: Answer key 


Note. SE = Sustaining Effort, GA = Getting Along with Others, MC = Maintaining Composure, 
KOM = Keeping an Open Mind, SC = Social Connection, N attempt = number of attempts, T 
spent = time spent, Var of T spent = variability of time spent. All correlations are statistically 
significant at p < .05. 
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Table 8. Correlations Between LMS Activities and Grade and SE Skills: Math 

LMS Activity Grade iS) GA MC KOM 
Number of Sessions 9 | | BaD. 

T spent: Worksheet 29 

T spent: Remediation makeup 22 

Var of T spent: Exam review materials 22 

Var of T spent: Worksheet 4 

T spent: Exam review materials 21 

T spent: Discussion board .20 

N attempt: Information -.19 

N attempt: Discussion board .19 

N attempt: Answer key Pal 4 

T spent: Lecture notes .16 

Var of T spent: Gradebook 15 

T spent: Answer key ANS 

N attempt: Reading guide sho 

N attempt: Learning objectives ANS 

T spent: Learning objectives 14 

Var of T spent: Information 3 .28 Foul 

T spent: General content .30 .36 .26 
T spent: Information Tf .28 22 
N attempt: Assignment -.26 -.22 
T spent: Assignment -.24 -.21 
N attempt: Information 23 23 22 
Var of T spent: General content .29 

Var of T spent: Remediation makeup 22 

T spent: Orientation Xe -.19 


Var of T spent: Assignment 
Var of T spent: Answer key 


Note. SE = Sustaining Effort, GA = Getting Along with Others, MC = Maintaining Composure, 
KOM = Keeping an Open Mind, SC = Social Connection, N attempt = number of attempts, T 
spent = time spent, Var of T spent = variability of time spent. All correlations are statistically 


significant at p < .05. 


SC 


.20 


-.24 
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Table 9. Correlations Between LMS Activities and Grade and SE Skills: Physics 


LMS Activity 

Number of Sessions 36 30 

T spent: Answer key ot Puli 

Var of T spent: Answer key Ol ANS 

N attempts: Answer key 24 15 

T spent: Lecture notes 22 AS) 

T spent: Exam review materials 21 5 

T spent: Gradebook .20 .20 

Var of T spent: Exam review materials .19 

N attempts: Reading quiz -.18 

Var of T spent: Gradebook Pal 4 

N attempts: Exam review materials a5) 14 a lke 

N attempts: Lecture notes 14 

N attempts: Orientation -.13 

T spent: Help resources 3 

Var of T spent: Exam questions Ail 

Var of T spent: Lecture notes Al 

T spent: Reading quiz 19 19 

T spent: Discussion board 18 14 15 
T spent: Gradebook si 13 
Var of T spent: Reading quiz AN .22 14 

T spent: Announcements si 14 

Var of T spent: General content .16 

Var of T spent: Announcements .16 

T spent: General content 14 

T spent: Email messages 14 ANS) NS 
Var of T spent: Email messages 14 6 
Var of T spent: Syllabus KS} 

T spent: Syllabus 18 

N attempts: Syllabus alld 

Var of T spent: Exam 14 

N attempts: Email messages 14 ANS 


Note. SE = Sustaining Effort, GA = Getting Along with Others, MC = Maintaining Composure, 
KOM = Keeping an Open Mind, SC = Social Connection, N attempt = number of attempts, T 
spent = time spent, Var of T spent = variability of time spent. All correlations are statistically 
significant at p < .05. 
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Table 10. Correlations Between LMS Activities and Grade and SE Skills: Psychology: 

LMS Activity Grade S) GA MC KOM 
Number of sessions ee oh) NG 

Var of T spent: General content .29 -.16 
Var of T spent: Chapter readings .28 

T spent: Exam .27 

Var of T spent: Online quiz .26 

T spent: Chapter readings .25 

T spent: Information 20) .20 
N attempts: Help resources 24 .30 

T spent: Textbook 24 18 

Var of T spent: Lab assignment 24 

T spent: General content .24 -.22 
Var of T spent: Textbook 23 

N attempts: Exam 25 

T spent: Gradebook .22 

Var of T spent: Gradebook P22 

Var of T spent: Exam 21 

Var of T spent: Chapter exam .20 

T spent: Syllabus .20 -.19 

Var of T spent: Extra credit 19 

T spent: Online quiz 18 

T spent: Extra credit ale 

Var of T spent: Announcements AT 

N attempts: Online quiz alld 

N attempts: Extra credit AT 

Var of T spent: Information 16 15 16 
T spent: Announcements 16 

Var of T spent: Help resources 16 2 ANS 
T spent: Help resources 16 19 

T spent: Lab assignment 16 

N attempts: Lecture class activities 15 

N attempts: Gradebook 215) 

N attempts: Textbook Pa re) 

Var of T spent: Lab activities sis) 

T spent: Lecture class activities 14 

N attempts: Announcements ale! 

Var of T spent: Discussion board 12 

Var of T spent: Orientation slZ 

N attempts: Discussion board 2 

T spent: Lab activities mala 

T spent: Discussion board bl 

N attempts: Email messages Ll 16 


Ie; 


23 
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Table 10. Correlations Between LMS Activities and Grade and SE Skills: Psychology—continued 


LMS Activity 

N attempts: Chapter exam -.20 

T spent: Email messages 19 NS 

Var of T spent: Email messages 18 

N attempts: Lab activities -.17 

N attempts: Chapter readings -.17 

N attempts: Orientation .16 16 

N attempts: Information 15 18 24 
N attempts: Clicker 21 

N attempts: Reading quiz -.16 


Note. SE = Sustaining Effort, GA = Getting Along with Others, MC = Maintaining Composure, KOM = 
Keeping an Open Mind, SC = Social Connection, N attempt = number of attempts, T spent = time spent, 


Var of T spent = variability of time spent. All correlations are statistically significant at p < .05. 


Table 11. Descriptive Statistics for Grades and Social and Emotional Skills 


Chemistry Math Physics Psychology 

M psy) SYD) M SYD) M psy) 
Grade Zk 11,0 2.19 | 27 2.70 | 22 Dan) 1.30 
Sustaining Effort -.09 95 -.25 .96 -.19 .90 -.21 91 
Getting Along with Others -.19 oS -.02 9) -.13 ge -.06 .86 
Social Connection -.18 .89 -.13 89 -.19 .86 -.18 94 
Maintaining Composure -.08 .88 10 88 -.13 89 -.04 .90 


Keeping an Open Mind -.22 92 -.11 95 -.17 91 -.15 92 
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